Skip to content

test(neb,confsearch): AUD-007 — pin what checkpoint resume really preserves - #82

Merged
hydrogen1222 merged 6 commits into
mainfrom
test/aud-007-resume-equivalence
Nov 14, 2026
Merged

hydrogen1222 merged 6 commits into
mainfrom
test/aud-007-resume-equivalence

Conversation

@hydrogen1222

@hydrogen1222 hydrogen1222 commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Tracking: #75 (Living code audit & exhaustive scientific test campaign)

AUD-007 — does checkpoint resume really keep structure, RNG and model identity?

Test-only. No production behaviour changed (one docs clarification, see below).

What was checked

Question NEB confsearch
structure restored? yes, verbatim yes, verbatim
RNG state restored? n/a — deterministic (fixed-seed pre-flight) yes — per-replica md_rng/mc_rng bit-generator states
model identity enforced? yes (resume fingerprint) yes (resume fingerprint)
prior evidence kept? yes (immutable step_* checkpoints) yes (incomplete_attempt_N/)

confsearch — exact equivalence

The engine checkpoints after every unit (checkpoints/confsearch.json) and stores
replica.md_rng.bit_generator.state / replica.mc_rng.bit_generator.state, then
restores them in _restore() before the resumed units run.

test_resume_equivalence.py kills a run after every durable unit boundary
(1–3 units for 1 replica / 3 cycles, and 1–4 units for 2 replicas / 2 cycles) and
asserts the resumed run's search_results.json is identical to an
uninterrupted run's — ranking, minima, diagnostics and all — with the single
exception of model.model_path (where a model is stored is not identity).

Mutation checks (each mutation makes the equivalence tests fail, so they are not
vacuous):

  • skipping the bit_generator.state restore → 2 failures;
  • perturbing the restored replica.current by 0.01 Å → 2 failures.

NEB — exact structure, geometry-only path

NEBCheckpointStore writes the whole band as an ASE trajectory plus a
band_sha256, and _read_checkpoint_directory verifies the checksum before
_validate_loaded_band re-checks atom count, indices, masses, PBC, cell and
symbols. The tests confirm:

  • the resumed attempt's first checkpoint is byte-identical to the band the
    interrupted attempt had checkpointed (structure restored verbatim);
  • the earlier attempt's step_* checkpoints and its exports survive the resume;
  • the resumed run converges (status=completed, converged=True) to the same
    climbing image and the same saddle.

NEB has no stochastic optimizer; the only randomness is the gradient pre-flight,
seeded from a module constant, so there is no generator state to checkpoint.

Known, documented limitation (not a bug): the checkpoint declares
geometry_resume_only: True, so a resume restarts the NEB optimisation from
the checkpointed geometry instead of replaying the optimiser state. In the test
fixture the barrier is 0.9592721 eV uninterrupted vs 0.9590986 eV resumed —
a ~2e-4 eV difference. The test therefore asserts the same saddle within a loose
1e-3 eV guard (explicitly not a precision claim). This was already recorded
in the checkpoint and run_context.json; this PR adds the missing sentence to
docs/workflows/neb.md / .zh.md so the contract is explicit for users too.

Mutation check: perturbing the restored band by 0.001 Å makes
test_resume_restores_the_checkpointed_band_exactly fail.

Evidence

DATA IMPACT

No scientific record invalidated (test-only + docs). No stored trajectory,
checkpoint or evidence file is touched.

Residual

  • Real-backend checkpoints were not exercised (synthetic calculators only).
  • test_resume_converges_to_the_same_saddle uses a synthetic curved MEP; it
    verifies the resume contract, not model accuracy.

AUD-007 (tracker #75).  No production behaviour changes; test-only plus one
documentation sentence.

confsearch: the engine already checkpoints the per-replica md_rng/mc_rng
bit-generator states after every unit and restores them in _restore().  The
new test kills a run after every durable unit boundary (1-3 units for one
replica / three cycles, 1-4 for two replicas / two cycles) and asserts the
resumed run's search_results.json is identical to an uninterrupted run's,
apart from model.model_path (not identity).  Verified non-vacuous: skipping
the RNG restore or perturbing the restored geometry each make it fail.

NEB: the checkpoint stores the whole band plus band_sha256, and the loader
verifies the checksum and re-validates the band.  The new tests show the
resumed attempt starts from a byte-identical band, that the interrupted
attempt's checkpoints and exports survive, and that the resumed run converges
to the same climbing image and saddle.  NEB has no stochastic optimizer, so
there is no generator state to restore.

The checkpoint is geometry_resume_only, so a resume restarts the optimiser
instead of replaying its state: in the fixture the barrier is 0.9592721 eV
uninterrupted vs 0.9590986 eV resumed (~2e-4 eV).  That limitation was already
recorded in the checkpoint and run_context.json; document it in
docs/workflows/neb.md and its Chinese twin as well.

DATA IMPACT: no scientific record invalidated (test-only + docs).

Tracking: #75

@hydrogen1222 hydrogen1222 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AUD-007 independent review — changes requested before merge (documentation correctness).

The new CPU tests are useful, and the branch CI passed. However, both docs/workflows/neb.md and .zh.md state that geometry-only resume "therefore converges to the same saddle". That is not guaranteed for general NEB landscapes: restarting FIRE without its optimizer state may converge to a different stationary band or may not converge within the remaining budget. The tests show one synthetic MEP converges to a nearby saddle (~1e-3 eV), not a universal guarantee.

Please revise EN/ZH to: "Restores the checkpointed band but restarts optimization; convergence, final saddle and barrier can differ; verify convergence and compare results before interpreting." Update PR wording accordingly. Also this branch is 5 commits behind current main 3d6ca2e; rebase on current main and rerun CI before merging. Remember selected result-JSON equality does not imply all extxyz minima are bitwise identical (see #83). No source algorithm changes needed.

hydrogen1222 and others added 5 commits November 6, 2026 22:56
…esume

DATA IMPACT: no scientific record invalidated.
DATA IMPACT: no scientific record invalidated.
Preserve existing test coverage and correct geometry-only resume caveat.

DATA IMPACT: no scientific record invalidated.
Revalidate against merged #83 confsearch checkpoint schema v2 and subsequent changes. Preserve the independent NEB resume-equivalence regression tests and bilingual caveat.

Test-only; scientific outputs remain untouched.
@hydrogen1222
hydrogen1222 marked this pull request as ready for review November 14, 2026 08:37
@hydrogen1222

Copy link
Copy Markdown
Owner Author

Responding to the changes-requested review on this PR (2026-11-05).

1. Documentation over-claim — fixed. docs/workflows/neb.md and
docs/workflows/neb.zh.md no longer say a geometry-only resume "therefore
converges to the same saddle". Both now say: the band recorded in the checkpoint
is restored, NEB optimisation restarts from there (geometry_resume_only), the
optimiser state is not replayed, so convergence, the final saddle and the
barrier can differ from an uninterrupted run — verify convergence and compare
the results before interpreting them — and every earlier checkpoint and export
is kept. The PR description was updated to match.

2. Rebased/merged onto current main and CI re-run — done. The branch now
includes 4ddd54bb (latest main, which contains #92). All checks are green on
the new head, including installer-real-nep and test (3.10)–test (3.13).

3. Acceptance evidence (independent run, not part of this PR's assertions).

AC1 — confsearch resume equivalence. Interrupted at every durable unit
boundary (1–3 units for 1 replica × 3 cycles; 1–4 units for 2 replicas × 2
cycles) and resumed; compared against an uninterrupted reference run:
search_results.json identical (except model.model_path); every minimum's
atomic coordinates
in minima/all_minima.extxyz identical (numbers, positions,
cell, PBC, energy, fmax, move_mask); minima/registry.json identical,
including each member's own positions_A — i.e. the #83 defect surface; and the
md/mc bit_generator states recorded at each unit index identical. Negative
controls (mutation of the resumed state) were all detected: re-seeding the
replica RNGs, a 1e-6 Å perturbation of a restored member, and a simulated #83
regression (members borrowing the representative geometry). So #83's fix is
intact and actively guarded.

AC2 — NEB resume correctness. The resumed attempt's first checkpoint is
byte-identical to the band the interrupted attempt checkpointed (band
sha256-verified, then re-validated for atom count/indices/masses/PBC/cell/
symbols); the interrupted attempt's step_* checkpoints and vasp_path*
exports survive; the resumed run reaches status=completed, converged=True,
the same climbing image and the same saddle. Per your note, the barrier is not
asserted to be strictly identical: the test guards only a loose 1e-3 eV bound,
because the optimiser state is not restored (observed difference for this fixture
≈2e-4 eV).

AC3 — regression. Full suite on this head: 2077 passed, 48 skipped, 0 failed
(the two main-side failures that were red before — the stale DPA torch 2.10.0
assertions and the unconditional --allow-existing in the installer — were
repaired separately in #92 and are not attributed to this PR).

4. On your #83 caveat. Agreed: the committed tests assert equality of the
selected result JSON, not that every raw extxyz minimum is bitwise identical,
and the PR does not claim otherwise. The acceptance run above additionally
compared all minima frames and found them identical, but that coordinate-level
check is an audit run, not a committed assertion.

Residual scope is unchanged: synthetic CPU calculators only; no real-backend
resume was exercised, and test_resume_converges_to_the_same_saddle verifies the
resume contract on a synthetic curved MEP, not model accuracy.

@hydrogen1222
hydrogen1222 merged commit ec28ba3 into main Nov 14, 2026
14 checks passed
@hydrogen1222
hydrogen1222 deleted the test/aud-007-resume-equivalence branch November 14, 2026 08:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant